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Although scholars and policy analysts both value 
quality-controlled collections of research results, they evaluate 
research according to different criteria. Nevertheless, an 
examination of a pessimistic fictional research experience and a more 
productive actual experience indicate that, although differences in 
focus between scholars and policy analysts may make the interaction 
of research collections and policy analysis difficult, a productive 
match is possible through emphasizing the strengths of each. To 
facilitate this effective interaction, policy analysts must 
appreciate the need for theoretical context and recognize that it 
often will be communicated in terms of specific empirical results not 

r data collectors must develop 
indexing and research facilities capable of handling diverse 
questions framed in different ways and of allowing users to browse in 
the collection; rapid retrieval of detailed information must be 
facilitated; and policy analysts must provide feedback to the 
scholars and collectors responsible for the research collections. 
Specific implications for REDUC (Red de Documentacion en Educacion 
para America Latina y el Caribe) are that system development should 
recognize differences between the system's contributors and users, 
that considerable effort should go toward developing a thoughtful 
user base, and that administrative and advisory mechanisms should 
maximize informal communication between the producers and users of 
educational research. (XM) 
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The Speci al Case of Research Collections and Policy Analysis* 



Gregory A. Jackaon 
Harvard University 



"...I cannot guarantee that I carry all the facts in my mind. 
Intense mental concentration has a curious way of blotting out 
what has passed. The barrister who has his case at his fingers' 
ends and is able to argue with an expert upon his own subject 
finds that a week or two of the courts will drive it all out of 
his head once more. So each of my cases displaces the last..." 

-Sherlock Holmes, in The Hound of the Baskervillea 



Faced with this problem, aficionados of Conan Doyle will recall, Mr. 
Holmes maintained an elaborate set of notebooks and file boxes, in which 
he recorded all potentially useful facts about crime, criminals, and 
potential crimes that appeared in London newspapers. On many occasions. 
Dr. Watson reports, criminals found themselves in custody because Holmes 
was able to recover and use important facts from his files as needed — 
facts that others, including Scotland Yard, found important only in 
retrospect . 

The essential features of Holmes's files were that collection 
according to specified rules preceded use, and that a carefully designed 



iThese remarks were prepared for a conference on REDUC (Red de 
Documentacidn en Educacidn para America Latina y el Caribe) held at the 
Organization of American States in Washington D.C., U.S.A., on October 
16 and 17, 1986. 
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index permitted acceaa tailored to the problem at hand. These same 
features define REDUC and similar collections of educational research 
intended, in part, to inform policy decision making. But such collec- 
tions differ from Holmes's in one key respect: whereas Holmes both 
collected and used his data, scholars (or others following scholars' 
procedures) place material into research collections for policy analysts 
and other decision makers to use. 

Scholars base current research on earlier conceptual and empirical 
work. To facilitate this they develop systems for publishing, archiv- 
ing, abstracting, indexing, locating, and retrieving earlier work. In 
the interest of parsimony such systems retain good work and discard the 
rest, and therefore require rules for evaluating research and maintain- 
ing quality. Such rules figure prominently, albeit implicitly, in 
mechanisms for creating collections of research results. 

Policy analysts, meanwhile, increasingly use tools capable of incor- 
porating research results into planning, particularly in education and 
other social domains. Such tools prove particularly useful when 
relevant, useful research results are readily available — when, for 
example, an existing collection of research results meets these quality- 
control criteria. Xt follows that both scholars and policy analysts 
value quality-controlled collections of research results. 

If scholars provide policy analysts with collections of research, 
and both groups value quality, it seems to follow that planning and 
decision making will improve* This rarely proves true, I will argue, 
largely because scholars and policy analysts evaluate the quality of 
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information differently. Therefore, scholars do not produce the 
collections policy analysts need, and policy analysts expect research 
collections scholars generally will not produce. Let r.e illustrate the 
argument I an instructed to be brief, which precludes full proof 
with a fictional, general, and somewhat pessimistic example, then 
recount a related and more productive experience in the United States 
(both dealing with enrollment policies in higher education), and finally 
try to identify some principles for an effective integration of schol- 
arly and policy interests in research collections. Without such 
integration, I submit, there can be little productive interaction 
between educational research and decision making. 

It Can't Work... 

Scholars review research according to principles established within 
various methodological and disciplinary domains. A quantitative 
sociologist, for example, might well evaluate studies of college 
enrollment according to their representation of relevant sociological 
theory — on status attainment, say, which includes educational attain- 
ment — and according to their use of statistical techniques such as 
regression analysis. Studies which score well according to both 
criteria "pass", and constitute journals, indexes, ERIC, and so forth; 
studies deficient in either or both "fail", perhaps to be revised and 
"passed" later. It is moat important, from the scholar's perspective, 
that a study be done and set "right"; otherwise the rules of scholarly 
discourse place it beneath notice. 



Research Collections and Policy Analysis 

Page 4 



Thus have there developed two major bodies of work on postsecondary 
educational choice. One draws on sociological theory and data to argue 
that social class and social context contribute mightily to early 
academic achievement, which joins its precursors to influence postsecon- 
dary choice. The other body of work draws on econoaic investment and 
consumption models to argue that social class, social context, and 
academic experiences jointly produce various tastes for education in 
individuals, and that to decide whether to enter university prospective 
students weigh the economic and other returns education brings, includ- 
ing the satisfaction of tastes, against its costs and the relative costs 
and benefits of other choices 2 . The relevant studies in both domains 
tend to rely on longitudinal surveys of student backgrounds and choices, 
using various statistical techniques for evaluating different variables' 
effects and interactions, or on detailed case studies of individual 
choices. These bodies of work are well documented in standard collec- 
tions of sociological, economic, and educational research. 

Now, to move from scholarship to policy analysis, consider a 
Minister of Education committed to increasing the number of citizens 



2 For reviews of these literatures as they bear on the United 
States, see 6. Jackson, "Public efficiency and private choice in higher 
education", Educational Evaluation and Policy Analysis 4 (1982) 237-47, 
or D. Terkla and 6. Jackson, "The state of the art in student choice 
research" (1984), soon to be available (a chronic problem) as an ERIC 
document. A classic exemplar of the sociological approach is to be 
found in V. Sewell, R. Mauser, and D. Featherman, Schooling and achieve- 
ment in American gflffilafcy. (New York: Academic Press, 1976), and of the 
economic approach in C. Hanski and D. Wise, College choice in Aperies 
(Cambridge HA: Harvard, 1983). The most recent work is my "Workable, 
comprehensive models of college choice", a 1986 report to the U.S. 
Department of Education not readily available except directly from me. 
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with bachelor's degrees, perhaps in economically important fields such 
as science, engineering, and agriculture. She wants to know how 
university students differ from other individuals of like age, and 
particularly how the Government can jnodify those differences and thereby 
(she hopes) increase enrollments. Is access the problem, in which case 
creating new institutions in underserved areas might help? Or should 
the Government work to reduce financial barriers at existing institu- 
tion*? What about curriculum reform, faculty salaries, and athletic 
facilities? 

An advisory panel of sociologists, using its established research 
data base, informs the Minister that an array of variables — social - 
class background, academic experience in high school, and the availa- 
bility of nearby colleges or universities, in particular work 
together to affect enrollment decisions; they provide her with abstracts 
of well-respected articles demonstrating just this in several different 
contexts. The Government Statistical Bureau, relying on its various 
data archives, provides tables of university-age individuals by social 
class, distributions of high-school grades, and maps of population and 
university locations. 

Everything appears to be working just fine. The data look impres- 
sive, and rich conceptual descriptions of educational choice abound; 
much of this has come from established research collections, some of 
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them spanning national boundaries^. After some thought, however, the 
Minister finds herself dissatisfied. She must choose among possible 
policies and justify her recommendations in terms of relative cost and 
benefit, but neither the sociologists nor the statisticians will make 
comparative projections: the sociologists require individual data on 
interactions among variables to use their statistical models properly, 
while the Statistical Bureau offers only aggregate data and avers that 
its job is to document, not to guess. Consulting economists yields the 
Minister different articles and concepts, but the same unsatisfactory 
result. 

Ultimately the Minister asks her staff and consultants to assemble 
useful data, however informal and ad hoc , so that she can make her 
decision and justify it. They do this through anecdotes, calls to 
administrators involved with student choice, and some organized guess- 
work. The Government's policy analysis draws only marginally on the 



3 It bears noting in this company that ERIC identifies a socio- 
logical counterargument baaed on Chilean data, where Ernesto Schiefel- 
bein and Joaeph Farrell have found that educational variables may 
influence aocial attainment more than family variables, and that 
educational quality ia more important than educational attainment. J. P. 
Farrell and E. Schiefelbein, "Education and atatua attainment in Chile: 
A comparative challenge to the Wisconsin model of atatua attainment" , 
Comparative Education Review 29 (1985) 490-506; E. Schiefelbein and J. P. 
Farrell, "Educational and occupational attainment in Chile: The effects 
of educational quality, attainment, and achievement", American Journal 
of Education 92 (19A4) 125-62; E. Shiefelbein and J. P. Farrell, Eight 
years of their lives: Throu gh schooling to the labor market in Chile 
(Ottawa: International Development Research Centre, 1982). The authors 
attribute the difference between Chile and the United States to the less 
stable socioeconomic structure in Chile, and the Minister might well 
want to consider her own country's situation as she proceeds. Identify- 
ing such cross-cultural variation is a major service research collec- 
tions can provide. 
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carefully collected wisdom of acholara in relevant disciplines, and 
extensively (but untraceably) on the assumptions, insights, "and preju- 
dices of numerous civil servants. The problem is that scholars' 
emphasis on design and focus meshes poorly with policy analysts' need 
for breadth and timeliness. 

Neither side is behaving unreasonably here; each seeks data which 
meet its criteria for quality. For acholara these involve theory and 
method, and lead to an emphasis on longitudinal or intensive research. 
But longitudinal research is almost by definition not timely, while 
intensive individual research, however insightful, is difficult to 
generalize to larger populations with any statistical accuracy. For 
policy analysts the criteria for useful research are timeliness, 
comprehensiveness, accessibility, and extrapolability . Studies which 
meet these criteria often gloss over important theoretical distinctions, 
blur individual or aubpopulation differences, and draw inferences about 
individual-level phenomena from aggregate data. However reasonably the 
aides behave, in view of these incompatibilities they probably will not 
collaborate effectively through a research collection. If my example 
represents reality, the prospects for ERIC, REDUC, policy-analysis 
journals, and similar services which seek to serve both audiences appear 
bleak, and educational research will have little influence on decision 
making*, 

4 Thia approximates the conclusion Jean-Pierre Vielle reached in his 
study of educational research and policy in Mexico, J. P. Vielle, "La 
capacidad y el impacto de la inveatigacidn educative" (Mexico: Programa 
Nacional Indicativo de Inveetigacidn Instructive, Reunionea de Informs- 
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. . .But It Can. . . 

Let me move quickly to a true story about higher-education policy 
analysis in the United States, one which augurs better for the role of 
research collection in policy analysis. During the mid-1970s the United 
States Congress expanded federal f inanciai-aid programs for post- 
secondary students and institutions, including scholarships, loans, 
various forms of loan guarantees, support for specific institutional 
activities, and in a few cases general institutional support. Combined 
with growth in the number of potential students, this produced sharp 
increases in federal expenditures on higher education. These expendi- 
tures clearly were headed for unsustainable levels, so budget cuts were 
inevitable. 

Since many of the federal financial-aid programs involved entitle- 
ments or guarantees it was difficult to estimate the savings various 
cuts sight entail or to compare these in light of their political costs. 
David Mundel, once a colleague of mine at Harvard but then working at 
the U.S. Congressional Budget Office, was asked to develop a simulation 
model capable of estimating the effects of likely changes — such as 
needs tests, institutional exclusions, award limits, debt limits, or 
emphasis on certain majors — on enrollment patterns by region, on 

cidn Educative, 1979, mimeo) — a conclusion strongly disputed by Carlos 
Huflo2 Iglesiaa in an editorial, Reviata Latlnoamerlcana de Estudioa 
Educatlvoa 10 (1980) v-xiv. Vielle and Hufloz were arguing about the 
relative merits of scholarly and action research, but the seme issues 
speak to the relationship between research collections and policy analysts. 
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overall program expenditures, and on the distribution of program 
benefits by region, socioeconomic and minority groupings, and so forth. 

Mundel began by consulting "the literature" — not, as it happens, 
through ERIC, but through an essentially equivalent search involving 
other indexes and bibliographies. His conclusion essentially matched 
the Minister's in the fictional example above: existing research was 
"good", in the scholarly sense, but provided too inconclusive a base for 
his assignment. Theories were, Mundel observed, more detailed and less 
inclusive than he required, and most statistical models were too 
specialized to be usable with existing demographic and distributional 
data. Controlling his instinct to abandon the scholarly approach and 
start from scratch 5 , he tried to extract specific effect estimates from 
the diverse studies. Matters looked up. On several points, such as the 
response of prospective students to price and financial aid and the 
effects of college or university proximity, he discovered great consis- 
tency across studies, particularly after adjustment for methodological 
differences. Moreover, the broad features of the postsecondary-choice 
process varied little from study to study, even though some studies 
ignored effects others found important. Gradually Mundel developed an 
outline of the choice process and a general sense of variables' relative 
impacts. 



^Mundel was, as it happens, co-author of a seminal work on college 
choice: M. Kohn, C. Hanski, and D. Mundel, "An empirical investigation' 
of factors which influence college-going behavior". Annals of Economic 
and Social Measurement 5 (1974) 391-419. This may explein his persistence. 
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Search-baaed model in hand, Mundel built a atatiatical model, 
bringing in available demographic data and specific estimates where 
possible, making reasonable assumptions in other cases, and repeatedly 
testing the sensitivity of his model's predictions to changes in 
assumptions. Before long he had developed an ad hoc model responding to 
the Congress's needs, and could assert safely thai it represented both 
current scholarly research and the specific policy interests which 
motivated it. Hundel's model eventually evolved into one used by the 
Department of Education, where it continues to influence policy; its 
longevity stems, in part, from a relatively sound theoretical base. 

If this story generalizes then the outlook for scholarly, policy- 
relevant archives like ERIC and REDUC is less bleak than it seemed in 
the preceding section. However, positive results seem to require a 
strong, thoughtful, judgmental, integrative user willing to evaluate and 
extend the product of research collections. In other words, whether a 
research collection is useful depends as much on the user as it does on 
the collection. This suggests a broadened emphasis on users among those 
seeking to advance research collections as policy-analysis tools, an 
emphasis generally absent in ERIC but, if this meeting is any guide, 
increasingly present in REDUC. Let me conclude with some comments on 
the appropriate relationship between users and research collections. 
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...if Everyone Knows What to Expects. 

Thua far I have concentrated on fictional and concrete atoriea 
illuatrating the difficult interaction between policy analyaia and 
reaearch collectiona and pointing to the requirements for auccesaful 
connection. Let me rehearse the three key points that emerge. 

Scholars a nd policy analysts evaluate research according to 
different criteria . 

Scholars determine whether given pieces of research meet accepted 
standards for theoretical foundation and research method in the appro- 
priate discipline, attending to (and "keeping", in the archival sense) 
studies that do and ignoring those that do not. Ambiguity arises around 
paradigmatic changes within disciplines, and around disciplinary 
differences. Policy analyets,- on the other hand, assess the relevance 
of research to their assignment (policy analysts virtually always are 
working on a legislative or executive assignment), its adequacy as a 
basis for projection (policy analysis very often involves projection), 
and its timeliness, often finding that only ad hoc research will do. 

Neither poli cy analysts nor scholars like to compromise th^ir 
notion of qu ality: each expects the other aide to admat . 

Thia stems not from childishness, but from each side's reasonable 

desire to do its 30b before it does someone else's. Incentives lie at 

the heart of the difficulty: policy analysts are rewarded for producing 

the required analysis, not for synthesizing scholarly works; scholars 

are rewarded for being scholarly, not for modifying disciplinary 
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conventions to meet someone else's needs. Exceptions appear only when 
individuals in either camp derive satisfaction from working effectively 
in the other: when policy analysts become interested in the theoretical 
context for their work, and in the accretion of knowledge over time, or 
when scholars become interested in the practical implications of their 
work for improving practice. 

A producti ve match between research collections and policy 
analysis in volves emphasizing the strengths of each: drawing 
broad, theoretically sound models and rough estimates from 
collections, and relying on ad hoc research to translate these 
into useful answers . 

Expecting scholarly research by itself to provide full, timely 
answers to practical questions or ad hoc policy analysis by itself to 
find robust theoretical ground is futile, but each is unexcelled on its 
own turf. Integrating the two domains, however, is the exclusive 
province of policy analysis, since the policy relevance of research 
cannot be discerned or evaluated until a question appears. This has 
implications for the necessary attributes of users and for search 
facilities, to which I now turn. 

...and Is Properly Equipped. 

I recounted David Mundel's experience above because it illustrates 
clearly what is necessary for research collections to function effec- 
tively and to improve policy analysis. 

Fir at* the policy analyst must appreciate the need for theoretical 
context, and recognize that this theoretical context often will be 
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communicated in terms of specific empirical results not themselves 
useful. 

Second , scholars and other data collectors must recognize their 
limited ability to foresee what questions will be asked of their data. 
They must develop indexing and search facilities capable of handling 
diverse questions framed in different ways, and of allowing interested 
users to "muck around" in the collection. (In open-stack libraries, 
readers may recall, this involves using the catalog to find the right 
floor and shelf, and then seeing what else is shelved nearby; research 
collections, particularly computerized ones, rarely provide analogous 
forms of access.) 

Third, it must be possible to retrieve detailed information from 
collections rapidly; one- or two-week intervals between computer 
searches for relevant database entries and the arrival of detailed hard 
copies, typical for educational -research collections, often exceed 
political tolerance. 

Fourth, and perhaps most important in the long run, policy analysts 
who manage to use research collections to good effect must provide 
feedback to the scholars and collectors responsible; this can provide 
not only guidance, but also the incentive otherwise lacking for schol- 
arly work to serve policy needs. 
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REDUC 

If this argument contains morals for REDUC they are that system 
development should recognize differences between the system's contribu- 
tors and users, that considerable effort should go to develop thoughtful 
uaera of its contents, and that administrative and advisory mechanisms 
should maximize informal communication between the producers and users 
of educational research. To.be avoided are designs which reduce system 
responsiveness, which presume that decision makers will change their 
behavior to match REDUC' a, or which evaluate quality according to either 
scholars' or policy analysts' criteria alone. 

There is little question educational research can inform and improve 
decision making. But there is little reason, given the political and 
social importance of educational decisions, to expect research — and 
particularly research collections such as REDUC -- to have overwhelming 
or immediate effects. Expecting too much, under these circumstances, 
entails receiving discouragingly little; hoping and designing for the 
best — but not expecting it! may well yield success. 
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